Universal Estimation of Time-varying Distributions

We investigate the estimation of distributions with timevarying parameters. We introduce an algorithm that achieves the optimal negative likelihood performance against the true probability distribution. We achieve this optimum regret performance without any knowledge about the total change of the parameters of true distribution. Our results are guaranteed to hold in an individual sequence manner such that we have no assumptions on the underlying sequences. Our log-loss performance with respect to the true probability density has regret bounds of O( √ CT ), where C is the total change (drift) in the natural parameters of the underlying distribution. We achieve this square-root regret with computational complexity only logarithmic in the time horizon, thus our algorithm is suitable for big data. Apart from the regret bounds, through synthetic and real life experiments, we demonstrate substantial performance gains with respect to the state-of-the-art probability density estimation algorithms in the literature.